Multi-Criteria Approaches to Markov Decision Processes with Uncertain Transition Parameters

Markov decision processes (MDPs) are a well established model for planing under uncertainty. In most situations the MDP parameters are estimates from real observations such that their values are not known precisely. Different types of MDPs with uncertain, imprecise or bounded transition rates or probabilities and rewards exist in the literature. Commonly the resulting processes are optimized with respect to the most robust policy which means that the goal is to generate a policy with the best worst case behavior. However, implementing such a policy could lead to potential losses of reward in most situations. In general, one is interested in policies which behave well in all situations which results in a multi-objective view of decision making. In this paper we consider policies for the discounted reward of MDPs with uncertain parameters. In particular, the approach is defined for bounded-parameter Markov decision processes (BMDPs) [GLD00]. In this setting the worst, best and average case performance of a policy is analyzed. The paper presents some theoretical results and algorithms to compute or approximate the convex hull of pure Pareto optimal policies in the value vector space.